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Abstract 


Probabilistic  planning  methods  can  have  various  objectives:  usually  they  either  max¬ 
imize  the  probability  of  goal  achievement  or  minimize  the  expected  execution  cost 
of  the  plan,  and  therefore  assume  that  the  agent  that  executes  the  plan  has  a  risk- 
neutral  attitude.  Although  there  are  many  situations  where  risk-sensitive  behavior  is 
more  appropriate,  researchers  have  largely  ignored  the  question  how  to  incorporate 
risk-sensitive  attitudes  into  their  planning  mechanisms.  Utility  theory  shows  that  it  is 
rational  to  maximize  expected  utility,  given  that  the  agent  accepts  a  few  simple  axioms 
and  has  unlimited  planning  resources  available.  Thus,  researchers  might  believe  that 
one  could  allow  for  risk-sensitive  attitudes  by  replacing  all  costs  with  their  respective 
utilities  (for  an  appropriate  utility  function).  We  show  that  this  is  usually  not  the  case 
and,  moreover,  that  —  in  general  —  the  best  action  in  a  state  can  depend  on  the  total 
cost  that  the  agent  has  already  accumulated  (that  is,  the  Markov  property  does  not 
need  to  hold).  However,  we  demonstrate  how  one  can  transform  planning  problems 
for  risk-sensitive  agents  into  equivalent  ones  for  risk-neutral  agents  provided  that  ex¬ 
ponential  utility  functions  are  used.  The  transformed  planning  problems  can  then  be 
solved  with  probabilistic  AI  planning  methods  or,  alternatively,  dynamic  programming 
methods.  If  the  agent  is  risk-seeking,  then  one  can  use  any  planning  method  on  the 
transformed  planning  problem  that  either  minimizes  (or  satisfices)  the  expected  execu¬ 
tion  cost  or,  alternatively,  maximizes  (or  satisfices)  the  probability  of  goal  achievement. 
The  better  a  plan  is  for  the  transformed  planning  problem,  the  better  it  is  for  the  origi¬ 
nal  planning  problem  as  well.  Thus,  one  can  extend  the  functionality  of  these  planners 
to  risk-sensitive  planning.  —  To  demonstrate  our  algorithms,  we  usc^  a  probabilistic 
planning  framework  (“probabilistic  decision  graphs’’)  that  can  ea,sily  be  mapped  into 
Markov  decision  problems.  It  allows  one  to  describe  probabilistic  effects  of  actions,  ac¬ 
tions  with  different  costs  (resource  consumption),  and  goal  states  with  different  rewards 
(goodness).  We  then  apply  probabilistic  decision  graphs  to  finding  opl,imal  plans  for 
risk-sensitive  agents  in  a  stochastic  blocks-world  domain  and  a  stochastic  path-planning 
domain. 


1  Introduction 


In  recent  years,  numerous  planning  methods  have  been  developed  that  are  able  to 
deal  with  stochastic  domains.*  Probabilistic  planning  methods  are  important,  because 
actions  in  the  real  world  usually  do  not  have  deterministic  effects.  This  uncertainty 
can  be  an  inherent  part  of  the  domain,  but  is  often  due  to  limitations  of  the  planner 
or  the  agent  that  executes  the  plan:  the  planner  might  have  an  insufficient  model  of 
the  domain,  or  the  agent  might  not  be  able  to  execute  its  actions  with  the  required 
precision. 

Not  all  probabilistic  planning  methods  have  the  same  planning  objective:  different 
planners  consider  different  plans  to  be  optimal  for  the  same  planning  problem.  In  this 
paper,  we  concentrate  on  three  possible  planning  objectives:  to  maximize  the  proba¬ 
bility  of  goal  achievement,  to  minimize  the  expected  execution  cost,  and  to  maximize 
the  expected  utility  of  plan  execution. 

In  some  domains,  it  is  impossible  to  determine  probabilistic  plans  that  are  guaranteed 
to  achieve  the  given  goal.  When  planning  in  such  domains,  one  is  usually  satisfied 
with  finding  plans  that  succeed  with  a  given  probability,  see  [Bresina  and  Drummond. 
1990]  and  [Kushmerick  et  al.,  199.3].  Even  if  a  plan  exists  that  achieves  a  goal  with 
probability  one,  time  might  not  permit  to  find  it.  Then,  planners  often  use  an  any-time 
approach:  they  increase  the  probability  of  goal  achievement  over  time  by  incrementally 
extending  the  plan  envelope. 

In  other  domains,  however,  it  is  easy  to  construct  plans  that  always  succeed  for  sure 
(at  leiist,  in  the  limit).  Then,  one  needs  a  criterion  for  choosing  among  these  plans. 
Such  a  metric  is  for  example  the  total  execution  cost  of  the  plans^:  One  quantifies 
the  “resource  consumption”  (for  example,  time,  energy,  or  money)  of  an  action  with  a 
single  real  number  that  depends  only  on  the  action,  the  state  it  is  executed  in.  anrl  the 
resulting  state.  Then,  the  total  execution  cost  is  defined  to  be  the  sum  of  the  resource 
consumption  costs  of  all  actions  executed  from  the  time  at  which  the  agent  is  started 
in  the  start  state  until  it  stops  in  a  goal  state  (given  that  it  obeys  the  plan). 

Since  the  total  execution  cost  of  probabilistic  plans  can  vary  from  plan  execution  to 
plan  execution,  almost  all  planning  methods  reported  in  tlie  literature  use  the  rrprrfnl 
total  execution  cost  as  ranking  criterion:  Out  of  all  plans  that  guaranty  to  achieve 
the  given  goal,  they  choose  the  one  for  execution  that  minimizes  the  expected  total 
execution  cost  (when  optimizing)  or,  at  least,  one  whose  expected  total  ex<'cnt.ion  cost 
is  smaller  than  a  given  number  (when  satisficing).  Thus,  th(*y  assume  that  th('  agent 
that  executes  the  plan  has  a  risk-neutral  attitude. 

'Examples  of  probabilistic  planning  methods  inrhi<lr  [Smith,  MISS],  [Hrc'sina  and  Drnimnond.  1!)!>0], 
[f,'hristiansen  and  (ioldberg,  1!)1)0],  [llHns.son  rl  al.,  I!(5M)].  [Koenig,  liMM],  [n<'an  rl  til.,  MliKl],  [Knsh- 
merick  el  al.,  and  many  others. 

"An  example  is  travel  planning.  When  planning  how  to  get  to  a  given  conb'n'nce,  one  can  ('a-sily 
devise  plans  that  are  always  successful  (at  lea.st  under  normal  circumstance's).  Possible  evaluation 
metrics  for  these  plans  includ«;  travel  cost  or  travel  time. 


However,  people  are  usually  not  risk-neutral  for  non-repetitive  planning  problems. 
Whereas  a  risk-neutral  agent  does  not  care  about  the  variance,  a  risk-seeking  agent 
(“gambler”)  is  willing  to  accept  a  plan  with  a  larger  expected  total  execution  cost  if 
the  uncertainty  is  increased  and  vice  versa  for  a  risk-averse  agent  (“insurance  holder”): 
If  a  plan  is  executed  only  once  (or  a  small  number  of  times),  then  —  among  all  plans 
with  the  same  expected  total  execution  cost  —  the  larger  the  variance  of  the  total 
execution  cost,  the  larger  the  chance  to  do  much  better  than  average.  Of  course,  the 
chance  to  do  much  worse  rises  as  well. 

Imagine,  for  example,  that  your  task  is  to  design  a  robot  for  the  annual  AAAI  robot 
competition,  where  it  has  to  complete  a  given  task  (for  example,  “find  the  coffee  pot”) 
in  as  short  a  time  as  possible.  You  want  the  robot  to  win  the  competition,  but  — 
in  case  it  loses  —  do  not  care  whether  it  makes  second  or  last  place.  You  know  that 
your  robot  is  not  much  faster  than  your  competitors’  robots,  maybe  even  a  bit  slower, 
but  cannot  assess  the  capabilities  of  the  other  robots  in  enough  detail  to  use  them  for 
determining  the  utilities  of  the  various  task  completion  times  of  your  robot.  In  this 
case,  you  probably  want  your  robot  to  take  chances,  and  thus  a  risk-seeking  attitude 
should  be  built  into  the  robot’s  planning  mechanism. 

On  the  other  hand,  there  are  also  many  situations  where  people  want  to  avoid  risk. 
The  plans  produced  by  planning  methods  should  reflect  the  risk-attitudes  of  the  people 
that  depend  on  them.  However,  researchers  have  largely  ignored  the  (|uestion  of  how  t  o 
incorporate  risk-sensitive  attitudes  into  their  planning  mechanisms.  In  fact,  the  term 
“decision-theoretic  planning”  is  often  incorrectly  equated  with  optimal  (meta-)planning 
for  a  risk-neutral  agent. 

It  is  possible  to  achieve  a  risk-sensitive  attitude  by  ranking  plans  not  according  to  their 
expected  total  execution  costs,  but  according  to  their  expected  total  execution  costs 
plus  or  minus  a  fraction  of  the  variances  (Filar  ti  hI.,  1989]  [Karakoulas.  199:1],  or  by 
searching  for  plans  whose  total  execution  costs  are  optimal  in  the  best  or  worst  case 
(“nature  acts  like  a  friend  or  enemy”)  (Wit.senhausen,  1966]  [lleger.  1992],  sw  also 
[Moore  and  Atkeson,  1993a].  However,  utility  theory  [von  Neumann  and  Morgenst.ern, 
1947]  —  a  subfield  of  decision  theory  —  |>rovides  a  normative  framework  for  making 
decisions  according  to  a  given  risk  attitude,  provided  that  the  agent  accepts  a  few  simple 
axioms  and  has  unlimited  planning  resources  available.  Its  application  to  planning 
problems  has  been  studied  by  [Ctzioni,  1991],  [llussell  and  Wefahl.  1991],  [lladdawy 
and  Hanks,  1992],  [Wellman  and  Doyle,  1992],  [C!oo<lwin,  1992],  and  others,  'riierefore. 
we  would  like  to  stay  witinn  this  framework. 

The  utility-theoretic  approach  encompa-sses  the  other  two  approaches:  Maximixing  t  in’ 
probability  of  goal  achievement  is  rational  accor<ling  to  utility  theory  if  the  agent  do«'s 
not  incur  action  costs,  but  receives  total  reward  for  achieving  a  goal  state'  atid 
total  reward  r,ian-;,<,ni  otherwise,  wlu're Miniinixiug  I  h«' exp«'<  (e'*l  leelal 
execution  cost  is  rational  according  to  utility  tlu'ory  if  t  he'  age'ut  is  risk-iu'ut  ral. 

In  the  following,  we  will  first  review  the  ba,sic  com-epts  of  utility  th(H)ry  and  iIk'U 
describe  a  planning  framework  (“probabilistic  decision  graphs”)  (hat  can  e'asily  Ix' 


mapped  into  Markov  decision  problems  and  of  which  cost-annotated  decision  trees 
(the  kind  that  are  used  in  utility  theory)  are  a  special  case.  It  allows  one  to  describe 
probabilistic  effects  of  actions,  actions  with  different  costs  (resource  consumption),  and 
goal  states  with  different  rewards  (goodness).  Next,  we  will  show  that  replacing  all 
costs  and  rewards  with  their  respective  utilities,  but  leaving  the  planning  medianism 
unchanged,  usually  leads  to  erroneous  results  and,  moreover,  that  —  in  general  — 
the  best  action  to  execute  in  a  state  can  depend  on  the  total  cost  that  the  agent  has 
already  accumulated  when  deciding  on  the  action.  However,  for  utility  functions  with 
a  certain  property,  we  will  demonstrate  how  planning  problems  for  risk-sensitive  agents 
can  be  transformed  into  equivalent  planning  problems  for  risk-neutral  agents,  that  can 
then  be  solved  with  dynamic  programming  methods  or,  alternatively,  probabilistic  .AI 
planning  methods.  The  better  a  plan  is  for  the  transformed,  risk-neutral  planning 
problem,  the  better  it  is  for  the  original,  risk-sensitive  planning  problem  as  well.  VVe 
will  show  that  this  approach  can,  at  least  for  risk-seeking  agents,  be  implemented  with 
any  AI  planning  method  that  minimizes  (or  satisfices)  the  expected  total  execution 
cost.  The  same  statement  holds,  perhaps  surprisingly,  for  all  planners  that  maximize 
(or  satisfice)  the  probability  of  goal  achievement.  Finally,  we  will  use  both  a  lilocks- 
world  and  a  path-planning  example  to  illustrate  our  ideas  and  show  how  the  optimal 
plan  depends  on  the  degree  of  risk-sensitivity  of  the  agent. 

The  purpose  of  this  paper  is  to  motivate  the  importance  of  risk-sensitive  planning  and 
introduce  our  approach  to  it  that  builds  on  previous  work  by  [Howard  and  Mathe- 
son,  1972]  in  the  context  of  Markov  decision  theory.  It  contains  a  summary  of  our 
results,  but  describes  the  ideas  only  informally.  Mathematical  derivations,  more  pre¬ 
cise  statements,  and  additional  qualifications  are  given  in  [Koenig  and  Simmons.  1994). 
Although  our  discussion  can  easily  be  applied  to  meta-planning  problems  if  planning 
resources  are  limited,  that  is,  the  theory  of  “limited  rationality”  [Horvitz,  1988]  [Rus¬ 
sell  and  Wefald,  I99l]  (for  example,  making  decisions  when  to  stop  relining  a  plan  and 
start  to  execute  it),  we  assume  throughout  this  paper  for  simplicity  that  the  agent  lia,s 
unlimited  planning  resources  available. 


2  utility  Theory 

In  this  section,  we  will  briefly  review  some  of  the  concepts  of  utility  theory  —  a  tli«H)rv 
that  explains  why  people  do  not  always  minimize  expecU'd  costs  or  maximiz«M‘xpect<‘d 
rewards.  Daniel  Bernotdli  studied  around  1778  the  following  scenario,  known  a,s  the 
“St.  Petersburg  paratlox’’:'^ 

Yon  can  participate  in  a  lottery  at  most  once.  I’lu'  loti  rv  involves 
the  successive  tossing  of  a  fair  coin  until  heads  appears  for  the  lirst  lim«\ 

If  heads  appears  on  tin*  dh  toss,  yon  win  2'  dollars.  What  is  t  he  dollar 

■'From  here  on,  wn  iiso  t.ln?  ■’rewards'’  an<l  “rosts"  as  follows;  Hi'wards  can  Ix'  positive  or 

negative  values,  lint  costs  are  always  negative  valin*s. 


amount  Cmax  that  you  are  willing  to  pay  at  most  for  a  ticket  that  allows 
you  to  participate  in  this  lottery? 


If  people  who  own  w  dollars  initially  (“their  wealth”),  choose  not  to  participate  in  the 
lottery,  then  they  maintain  their  assets.  On  the  other  hand,  if  they  buy  the  ticket  for  c 
dollars  and  then  play  the  lottery,  they  own  2'+w—c  dollars  afterwards  with  probability 
1/2*.  Thus,  the  expected  reward  of  the  lottery  is  JZSi  =  oo  dollars  afterwards. 

No  matter  how  large  the  price  of  the  ticket,  the  expected  reward  of  participating  in  the 
lottery  is  larger  than  the  expected  reward  of  abstaining.  Since  we  assume  in  this  paper 
that  people  prefer  a  larger  reward  over  a  smaller  one,  they  should  be  willing  to  wager 
all  their  assets  if  they  were  indeed  maximizing  expected  rewards.  In  reality,  however, 
most  people  offer  only  around  ten  dollars  for  the  ticket. 

Bernoulli  suggested  that  people  do  not  average  over  rewards,  but  over  the  pleasure 
(“utility”)  that  the  rewards  provide.  For  every  risk-attitude,  there  exists  a  strictly 
monotonically  increasing  utility  function  that  transforms  rewards  c  into  real-valued 
utilities  «(c)  such  that  it  is  rational  to  maximize  expected  utility,  given  that  the  decision 
maker  accepts  a  few  simple  axioms  and  has  unlimited  planning  resources  available  [von 
Neumann  and  Morgenstern,  1947). 

A  lottery  is  recursively  defined  to  be  either  a  reward  that  is  received  for  sure  (that 
is,  with  probability  one)  or  a  probability  distribution  over  lotteries.  If  T  is  a  lottery, 
then  we  use  u[L]  to  denote  the  expected  utility  of  the  lottery  (or,  in  later  sections,  the 
expected  utility  of  a  state). u“‘(«[L])  is  called  the  certainty  (monetary)  equivalent  of 
lottery  L.  Maximizing  expected  utility  is  therefore  equivalent  to  maximizing  certainty 
equivalents. 

The  utility  of  not  participating  in  the  St.  Petersburg  lottery  is  «(»/;).  where  u  is 
the  utility  function  of  the  decision  maker.  Participating  in  t  he  lottery  results  in  the 
expected  utility  •  Thus,  people  should  wager  at  most  c„,ax  dollars,  where 

u{w)  =  •  The  following  table  shows  c,„„j.  depending  on  w  for  people 

with  utility  function  u(c)  =  logjC  (for  c  >  0): 


to 

1.00 

2.82 

10.00 

1.97 

100.00 

7.79 

1000.00 

10.95 

10000.00 

1  1.21 

100000.00 

17.55 

These  nutnlxTs  appc'ar  to  be  in  the  right  range,  although  Bernoulli  did  nol  claim  that 
people  have  this  particular  utility  function. 

'’Note  the  subtle  difTerenre;  »()  maps  costs  into  utilities,  atui  tiQ  trausronus  lotf.iTic's  (or  states) 
into  expectefi  utilities.  If  lottery  I,  contains  n'wani  c,  with  probability  (hen  m(/,]  = 


I 
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3U''((l-p)ll(C|)'fpll(C2)) 

Figure  1:  A  Concave  Utility  Function 


People  are  not  willing  to  pay  more  for  the  ticket,  because  they  are  risk-averse.  To  them, 
losing  parts  of  their  stake  has  more  weight  than  winning  a  fortune.  More  formally, 
people  are  called  risk-averse  (risk-seeking,  risk-neutral)  if  and  only  if  their  expected 
utility  of  every  non-deterministic  lottery  L  is  smaller  than  (larger  than,  equal  to)  their 
utility  of  the  expected  reward  E[L]  of  the  same  lottery.  (People  who  are  not  risk-neutral 
are  called  risk-sensitive.)  If  risk-averse  people  have  to  decide  between  participating  in 
an  arbitrary  non-deterministic  lottery  L  and  receiving  its  expected  reward  E[L]  for 
sure,  they  would  prefer  the  latter  option.  In  order  for  them  to  be  indifferent  between 
the  two  alternatives,  the  expected  reward  of  the  first  lottery  must  be  larger  than  the 
value  of  the  reward  of  the  second  lottery.  In  other  words,  for  risk-averse  people,  the 
certainty  equivalent  CME[L]  of  every  non-deterministic  lottery  L  is  smaller  than  its 
expected  reward  E[L]. 

People  are  risk-averse  (risk-seeking,  risk-neutral)  if  and  only  if  their  utility  functions 
are  concave  (convex,  linear).  Figure  1  illustrates,  for  example,  why  risk-averse  people 
have  concave  utility  functions.  It  shows  for  a  lottery  L  with  two  rewards  ci  (won  with 
probability  1  —  p)  and  cj  (won  with  probability  p)  that  iiuleed  u[L]  <  u{E[L])  and 
CME\L]  <  E[L]. 

Since  utility  functions  encode  the  individual  risk- preferences  of  pcH)ple,  tlu'y  must  lx* 
elicited  from  the  decision  maker  on  an  individnal  ba,sis.  Since  utility  functions  an* 
determined  only  up  to  arbitrary  strictly  positively  linear  transformations.  t>n<'  can 
establish  an  arbitrary  scale  by  fixing  the  utilities  of  two  rewar<ls.  say  (he  utility  of 
the  smallest  possible  rewarrl  C|  and  the  one  of  the  largest  possibh*  r('war<l  c^.  Assnux* 
ii(ci)  =  0  and  ufcj)  =  1.  To  determine  the  utility  of  any  reward  c,  one  a,sks  the  d<*cision 
maker  for  the  probability  p  that  makes  him/her  indifferent  betwwn  the  following  two 
lotteries:  Either  s/he  receives  reward  c—w  for  sure,  or  s/he  r('<‘eives  reward  C|  —  w  with 


probability  l—p  and  reward  Cj  —  u;  with  probability  p,  where  w  is  the  initial  wealth  of 
the  person.  Then,  w(c)  =  u(c  —  «;  + «;)  =  (!  —  p)u(ci  —  w  +  10)  +  pu(c2  —  w  +  w)  =  p. 


3  The  Planning  Framework 

The  following  representation  of  probabilistic  planning  problems  was  used  in  [Koenig, 
1991].  A  similar  framework  has  recently  been  used  by  [Dean  et  aL,  199.3]  and  is  com¬ 
monly  used  for  table-based  reinforcement  learning.  Although  we  use  this  particular 
planning  framework  here,  our  method  for  integrating  risk-sensitivity  into  planning  ap¬ 
plies  to  all  planning  frameworks  that  associate  “action  costs”  with  action-outcome  pairs 
such  that  the  execution  of  a  plan  can  be  evaluated  according  to  the  sum  of  the  “action 
costs”  of  the  executed  actions  and  their  outcomes  (“time-additive  value  functions”).  It 
Ctui,  for  example,  be  used  in  conjunction  with  Smith’s  planner  [Smith,  1988]. 

Planning  is  done  in  a  state  space.  S  is  its  finite  set  of  states,  .sq  €  S  the  start  state,  and 
G  C  5  a  set  of  goal  states.  A  plan  determines  at  every  point  in  time  which  action  the 
(artificial)  agent  has  to  execute  in  its  current  state.  In  a  goal  state  .s,  the  agent  receives 
a  (positive  or  negative)  one-time  reward  (“goal  reward” )  r[.s]  and  then  has  to  terminate. 
The  goal  rewards  reflect  that  different  goal  states  can  be  of  different  value  to  the  agent. 
However,  to  keep  the  following  discussion  simple,  we  will  use  only  planning  examples 
for  which  all  goal  rewards  are  zero.  In  a  non-goal  state  s,  the  agent  can  choose  an 
action  a  from  a  finite  set  of  actions  A{s).  Nature  then  determines  the  outcome  of  the 
action  with  a  coin  flip:  with  transition  probability  p“[s,s'],  the  agent  incurs  a  negative 
reward®  (“action  cost”)  c‘*[s,s']  <  0  and  is  in  successor  state  s'.  Thus,  we  assume  in 
this  paper  that  the  outcomes  of  all  action  executions  are  mutually  independent  given 
the  current  state  of  the  agent  (Markov  property).  The  action  costs  reflect  the  prices 
of  the  consumed  resources,  for  example  time  needed  or  effort  spent.  We  assume  t  hat 
the  values  of  all  parameters  are  completely  known  and  do  not  <iiange  over  time.  We 
do  not  assume,  however,  that  the  planner  uses  a  planning  approach  that  operates  in 
the  state  space  (instead  of,  say,  the  space  of  partial  plans). 

For  a  given  plan,  we  define  the  probability  of  goal  achievement  of  state  .s  as  the  proba¬ 
bility  with  which  the  agent  eventually  reaches  a  goal  state  if  it  is  started  in  .s  and  t)l>eys 
the  plan.  If  this  probability  equals  one,  we  say  that  the  plan  solves  .s.  A  plan  that 
solves  the  start  state  is  called  admissible.  In  the  risk-neutral  case,  a  plan  is  <'valuate<l 
according  to  the  expected  total  reward  of  the  start  state.  The  expect, imI  (,o(,al  reward 
y[.9]  (“net  profit”)  of  state  s  for  a  given  plan  is  the  t'xpected  sum  of  tin*  r<nvard  of  the 
goal  state  (measured  in  some  unit)  an<l  the  total  cost  of  the  actions  (nu'a.snr<'«l  in  t  he 
same  unit)  that  are  execiite<l  from  the  tiim*  at  which  the  agiMit  is  startl'd  in  s  until  it 
stops  in  a  goal  state  (given  that  it  obeys  the  plan).  Similarly,  llu'  «'xpe<M.e«l  total  utility 

■'’The  assiimpt.ion  «']  <  ()  can  ho  woakonoH.  Wo  ;ulc>|>l.  it  tliroiiglioiit  this  |))i|)<t  for  two  r<';».soiis: 
First,  it  is  consi.stont  with  goal  oriented  planning  prnhh'ins,  sinro  one  h:is  to  put  i  ffor!  into  aohioving 
a  goal.  The  goal  itself  is  the  reward.  Second,  it  is  a  simple  ami  siilficiont  romlition  for  hc'ing  ;d>lo  (o 
guarantee  that  an  agent  that  maximizes  oxp<Tto<l  total  reward  should  imh'od  I  ry  In  nvveh  a  goal  state. 
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u[s]  of  state  s  is  the  expected  utility  of  the  sum  of  the  goal  reward  and  the  total  cost 
of  the  executed  actions. 

The  planning  framework  described  above  is  very  general.  For  example,  one  can  easily 
represent  goal  states  in  which  the  agent  does  not  have  to  stop  (that  is,  that  the  agent 
can  leave  in  order  to  reach  a  different  goal  state  that  has  a  larger  goal  reward).  This 
is  necessary  if  one  wants  unsolved  states  to  have  an  expected  total  execution  cost  that 
is  finite  instead  of  minus  infinity.  One  could,  for  example,  allow  the  agent  to  stop  in 
any  state,  but  penalize  it  for  stopping  in  a  non-goal  state.  ( In  this  case,  all  non-goal 
states  must  be  converted  to  goal  states  that  have  a  very  small  goal  reward  and  can  be 
left  again.) 

For  risk-neutral  agents,  the  planning  framework  is  isomorphic  to  Markov  decision  prob¬ 
lems  [Mine  and  Osaki,  1970] :  Assume  that  a  planning  problem  is  given.  We  distinguish 
the  parameters  of  the  corresponding  Markov  decision  problem  by  adding  a  hat.  For 
example,  we  use  S  for  the  set  of  states  of  the  Markov  decision  problem.  Then,  the 
planning  problem  corresponds  to  the  following  Markov  decision  problem,  where  s  and 
a  are  two  arbitrary  symbols  with  s  ^  S  and  a  ^  U*e5\G^(-^)-  Every  state  .s  of  the 
Markov  decision  problem  that  corresponds  to  a  goal  state  of  the  planning  problem 
has  exactly  one  applicable  action,  namely  a.  The  execution  of  this  action  causes  the 
Markov  process  to  receive  reward  r[s]  and  change  state  to  state  s.  .Action  a  is  the  only 
action  applicable  in  state  s.  It  has  action  cost  zero  and  leads  deterministically  back 
to  s.  Once  the  Markov  decision  process  has  reached  state  .5,  it  can  no  longer  leave 
the  state  and  its  expected  total  future  reward  is  zero.  To  summarize,  when  the  agent 
reaches  a  goal  state  s  of  the  planning  problem,  it  receives  total  future  reward  r[.s].  This 
corresponds  to  the  Markov  process  reaching  state  .s  of  the  Markov  decision  problem, 
in  which  case  its  total  future  reward  is  r[.s]  as  well. 


S  :=  5U{.5} 


II 

if  s  €  S  \  G  ~  II  c 

•r  for  all  .-5  €  .S 

if  s  €  G  U  1 

if  €  .S'  \  G  and  .s'  €  S 

:=  -  2 

i  f  .s  €  .S  \  G  and  .s'  =  .s  , 

if  .s  €  G'U  {.?}  ami  .s'  €  ^  ^  ^ 

.  1 

if  .s  €  G  U  {.s}  and  .s'  =  .s 

f  cl-s,-**'] 

if  .s  E  S  \  G  ami  .s'  €  S 

c“[.s,s']  :=  1  r[.s] 

if  .s  €  G  and  .•*'  =  .s  for  all  .s.  .s'  g  .s'  and  a  €  .  M.-^) 

1  0 

if  .s  =  .s  and  .s'  =  s 

Although  the  planning  framework  is  isomorphic  to  Markov  (hrision  probh-ms  ami  «-an 
therefore  be  described  with  a  S'l'Ill  PS-like  notation  [Koc'iiig,  I99l]'\  we  n.s«>aii  <'a.si<'r-lo- 
depict  representation  here  (“probabilistic  decision  graphs”)  that  resembles  tiu'  kind  of 

'’For  the  original  STRIPS-notation,  that  applio.s  only  to  (l('t<'riniiiistir  doniains,  stH'  [Fik«'s  and 
Nilsson,  IU7l]. 


non-goal  state 


Figure  2:  Building  Blocks  of  Probabilistic  Decision  Graphs 


decision  trees  that  are  used  in  utility  theory.  Its  building  blocks  are  shown  in  Figure  2. 
Every  state  corresponds  to  a  (large)  circle.  The  large  circle  of  a  non-goal  state  .s 
contains  a  decision  tree  that  consists  of  a  decision  node  (square)  followed  by  chance 
nodes  (small  circles),  one  for  every  a  €  /l(.s).  For  every  outcome  of  an  action,  one 
specifies  its  transition  probability  and  action  cost.  The  circle  of  a  goal  state  contains 
a  value  node  (diamond)  for  the  goal  reward. 

To  represent  a  planning  problem,  these  building  blocks  have  to  be  connected  so  that 
there  are  no  dangling  outcome  arrows.  In  addition,  the  start  state  is  marked  with  an 
incoming  arrow  that  has  no  source  state  and  is  labeled  “start.*’  As  an  illustration,  con¬ 
sider  a  simple  planning  problem  in  a  blocks-world  with  three  indistinguishable  blocks. 
In  every  blocks-world  state,  one  can  move  a  block  that  has  a  clear  top  onto  either  the 
table  or  a  different  block  with  a  clear  top.  Moving  a  block  takes  one  minute.  With 
probability  0.1,  the  moved  block  ends  up  at  its  intended  destination.  With  probability 
0.9,  however,  the  gripper  loses  the  block  and  it  ends  up  directly  on  the  table.  (Thus, 
moving  a  block  to  the  table  always  succeeds.)  In  such  a  probabilistic  blocks-world, 
it  is  easy  to  determine  at  least  one  admissible  plan  for  every  solvable  planning  prob¬ 
lem:  First,  unstack  every  block.  (Remember  that  moving  a  block  to  the  table  always 
succeeds.)  Then,  stack  blocks  as  neederl  for  the  goal  configuration.  (If  a  stacking  ac¬ 
tion  fails,  repeat  it  until  it  finally  succee<ls.)  Figure  :l  shows  the  r<‘presentation  of  the 
planning  problem  to  unstack  fully  a  stack  of  three  blocks."  I''igure  I  contains  on<*  par¬ 
ticular  state-action  mapping  (“|)lan’’)  that  solves  the  planning  problem.  A  stat<'-a.ction 
mapping  (“stationary,  deterministic  policy”)  specifies  for  every  staU*  the  action  that 
the  agent  has  to  execute  when  it  is  in  tliat  state.  l'*or  Markov  dt'cision  probh'ins.  oih' 
can  restrict  plans  to  state-action  mappings  without  losing  optimality.  I''igur<*  I  also 

^The  symbols  “X”  and  “Y”  arc  introdncwl  only  to  enable  ns  lo  express  the  artions.  They  do  nol 
mark  specific  blocks  an<l  can  therefore  be  uswl  to  denote  differtnit  blocks  in  dilfen'iit  stag's. 


8 


iitan 


Figure  3:  A  Probabilistic  Decision  Grapli  (“Planning  Problem”) 


introduces  the  shorthand  notation  that  we  will  use  in  this  paper  to  represent  plans. 

Note  that  probabilistic  decision  graphs  can  have  cycles:  cycles  do  not.  imply  that  a 
decision  depends  on  itself,  but  that  a  decision  <lepends  on  t  he  same  decision  made  at 
an  earlier  point  in  time.  In  the  following,  we  will  distinguish  two  simplifications  of  this 
planning  framework,  namely  acyclic  probabilistic  decision  graphs  an<l  the  (‘wn  simpler 
acyclic  probabilistic  decision  graphs  toilhout  shaird  suhliTvs  (“cost-annotated  decisiojj 
trees”).  The  last  two  varieties  are  commonly  used  in  dc*cision  theory.  I'or  example. 
Figure  5  represents  the  planning  problem  to  «lecide  ImMavchmi  “buying  a  tickc't.  for  the 
St.  Petersburg  lottery  for  c  dollars”  or  “abstaiiiiug  from  tiu'  lottc'ry"  as  au  acyclic 
probabilistic  decision  graph  without  shared  subtrcvs  and  without  action  costs  (that  is. 
all  action  costs  art*  /(to).** 

'’Note  that  the  tiecision  tree  lias  —  in  contrast  to  the  plnniiiiif!;  frami'work  outlined  ahove  an 
action  with  an  infinite  niitnher  of  ontcoines. 


!) 


shorthand  notation 
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Figure  4:  A  State-Action  Mapping  (“’Plan”) 


4  The  Problem 

We  suspect  that  researchers  have  largely  ignored  the  question  of  how  to  incorporate 
risk-sensitive  attitudes  into  their  planning  mechanisms,  because  they  assume  that  by 
replacing  all  costs  and  rewards  with  their  respective  utilities  (for  an  appropriate  util¬ 
ity  function)  one  can  achieve  risk-sensitive  attittides  without  changing  their  planning 
mechanisms.  In  the  following,  we  will  use  acyclic  probabilistic  decision  graphs  to 
demonstrate  that  this  is  not  necessarily  the  case,  but  first  we  will  review  how  to  use 
dynamic  programming  techniques  to  determine  optimal  plans  for  risk-neutral  agents. 


4.1  Planning  for  Risk-Neutral  Agents 

A  risk-neutral  agent  has  to  solve  planning  task  PTl:  giv<'n  a  «  oniph'le  sp«'ei(ication 
of  the  planning  problem,  find  a  plan  for  which  the  start  state  has  the  largest  <‘xpecl<‘d 
total  rewarfl. 
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First,  we  would  like  to  point  out  that  every  planning  task  PTl  can  be  transformed  into 
an  equivalent  planning  task  that  has  only  one  goal  state,  which  has  goal  reward  zero. 
Planning  task  PTl  is  equivalent  to  the  following  planning  task  PTT  for  a  risk-neutral 
agent: 


Introduce  a  new  goal  state  with  goal  reward  zero,  and  replace  every 
(old)  goal  state  s  ^  G  with  a  non-goal  state  in  which  only  one  action 
can  be  executed,  that  leads  with  probability  one  and  action  cost  r[.s]  — 
maXj'gG  r[s']  — 1  <  0  to  the  new  goal  state.  Otherwi.se,  the  planning  problem 
remains  unchanged. 


Planning  task  PTl’  satisfies  all  requirements  of  our  planning  framework.  It  is  ecpii  valent 
to  planning  task  PTl  in  the  following  sense:  the  better  a  plan  is  for  planning  task 
PTl,  the  better  it  is  for  planning  task  PTl’  as  well,  and  vice  versa.  This  is  the  cas<*. 
because  the  expected  total  reward  of  every  admissible  plan  for  planning  task  PTl*  is 
maxygG  r[.s']-|- 1  lower  than  the  expectetl  total  rewartl  of  the  same  plan  for  planning  task 
PTl.  An  inadmissible  plan  has  expecterl  total  rewa.r<l  minus  infinity  for  both  planning 
ta.sks.  (Note  that  a  plan  is  either  admissible  for  both  planning  tasks  or  inadtnissible 
for  both  of  them.) 

An  optimal  .state-action  mapping  for  planning  task  P'l'l  can  be  dt'b'rmined  in  polyiu)- 
mial  time  with  dynamic  programming  techni<iues  (“Markovlian)  <l(H-ision  algorithms"). 
As  an  example  for  how  to  solve  a  given  planning  task  PTl  for  aryclir  probabilistic  cU'- 
cision  graphs  consider  the  (only  partly  specifie<l)  example  shown  itt  l'’igim'  (».  I'o  solve 


II 


sun 


Figure  6:  A  Planning  Problem  with  a  Shared  Subtree 


this  cost-annotated  decision  tree,  we  could  transform  it.  First,  we  propagate  the  action 
costs  to  the  value  nodes.  This  amounts  to  duplicating  shared  sid)trc?es,  since  every  path 
from  the  start  state  to  a  goal  state  needs  to  have  its  own  value  node.  The  resulting 
decision  tree  can  then  be  solved  in  time  linear  in  its  size,  as  shown  in  Figure  7:  the 
expected  total  reward  of  a  value  node  is  the  goal  reward,  the  expected  total  reward 
of  a  chance  node  is  the  average  over  the  ex|)ect,ed  total  rewards  of  its  successor  nodes 
weighted  with  the  transition  probabilities,  and  the  expected  total  reward  of  a  decision 
node  is  the  maximum  of  the  expected  total  rewards  of  its  successor  nodes.  The  action 
that  achieves  the  maximum  is  the  optimal  decision  for  the  decision  node,  riu’  (wptM'U'tl 
total  reward  r[.s|  of  a  (non-goal)  state  .s  e<|uals  the  <’xp<'ct.<Ml  total  r('war<l  of  t  he  decision 
no<le  that  it  contains,  and  the  optimal  action  n.[.s]  for  tin*  state'  is  the'  action  that  is 
optimal  of  its  ehxMsion  nodi'.  Actions  that  ari'  snb-opt.imal  are  "crossed  out.”  with  two 
horizontal  line's  in  the  figures. 

The  transformation  outlined  above  can  be  elone'  in  linear  time  if  no  snbtre'i's  are'  share'll. 
However,  if  subtrees  are  shareel,  it  is  expe’nsive,  since  t.he  number  of  paths  anil  t  hi'ii'- 
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Figure  7:  Solution  for  a  Risk-Neutral  Agent  I 


fore  the  size  of  the  transformed  decision  tree  —  can  be  exponential  in  the  number  of 
states  of  the  original  tree.  Fortunately,  it  is  well  known  that  the  following  dynamic 
programming  technique  (“[averaging-out-and-]folding-back”)  solves  acyclic  probabilis¬ 
tic  decision  graphs  for  risk-neutral  agents  on  the  original  trw  in  linear  time,  that  is. 
without  duplicating  shared  subtrees,  as  shown  in  Figure  8.  Dynamic  programming 
algorithms,  such  as  this  one,  can  be  used  to  solve  planning  task  PTl,  because  the 
Markov  property  holds  for  all  states:  the  expected  total  reward  e[.s]  of  every  state  (and 
thus  the  optimal  action  a[.s]  for  the  state)  is  iiulepeinlent  of  how  t  he  agent  r<’achecl  the 
state. 


r  1 _ (  »■[•'<]  for  s  e  a 

—  I  inax,g,,(,)E,'€.v + '’Kl)  <><l'<'i'wise 

rt(.sl  :=  argmax„€^(,)  +  e[.s'])  for  .seS\(! 

V6.S 


i:{ 


Thus,  one  evaluates  every  subtree  only  once  and  the  run-time  of  the  algorithm  is  linear 
in  the  size  of  the  original  decision  tree. 


4.2  Planning  for  Risk*Sensitive  Agents 


A  risk-sensitive  agent  has  to  solve  planning  task  PT2:  given  a  utility  function  and 
a  complete  specification  of  the  planning  problem,  find  a  plan  for  which  t  he  start  st.at<‘ 
has  the  largest  expected  total  utility.®  As  outlined  earlier,  planning  ta.sk  PT2  can 
be  solved  for  probabilistic  decision  graphs  without  action  costs  by  first  replacing  all 
goal  rewards  with  their  respective  utilities  and  then  using  any  planning  im'thod  for 
risk-neutral  agents,  riiis  is  illustrated  for  the  St.  Petersburg  paradox  in  Pigure 

In  reality,  however,  the  probabilistic  decision  gra|>hs  of  planning  (ask  P'P'J  do  hav<’ 


'The  expected  total  utility  h[«]  of  a  state  .s  for  a  given  plan  is  the  expert«>d  utility  of  tin-  (otal 
reward  (that  is,  goal  reward  plus  action  costs)  received  when  starting  in  s  and  ('xecnting  the  plan.  VV<' 
have  to  use  «[«]  instead  of  «(s),  since  starting  in  «  correspoinis  lo  participating  in  a  lottery. 
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Figure  9:  Solution  for  a  Risk-Sensitive  Agent 


action  costs.  Similarly  to  how  we  proceeded  earlier  for  risk-neutral  agents,  we  could 
first  propagate  the  action  costs  to  the  value  nodes  (which  involves  duplicating  shared 
subtrees  if  they  exist).  Next,  all  rewards  at  the  value  nodes  are  replaced  with  their 
respective  utilities.  Finally,  folding-back  is  used  to  determine  an  optimal  plan.  Re¬ 
member  that  this  method  has  an  exponential  run-time  in  the  worst  case  (and  is  not 
directly  applicable  to  cyclic  probabilistic  decision  graphs).'*’  Consitler  for  example  t  he 
planning  problem  for  a  risk-neutral  agent  again  that  was  shown  in  Figure  7.  riu'  <'or- 
responding  planning  task  for  a  risk-seeking  agent  with  utility  function  u(c)  =  — 

(for  c  <  0)  is  shown  in  Figure  9. 

It  is  not  optimal  to  simply  replace  all  costs  and  rewa.r»ls  with  tlu'ir  rt'spectivf'  utilities, 

"’For  acyriir  f>rohabilist.ir  (lorision  graphs  and  some  iit.ilily  riinrtion.s  oik'  ran  (  ry  l<>  <lo  iniirli  hri  lor 
by  rcmenibrring  for  every  sl.a(.o  the  fniirtional  depemleiiry  bel  wfs'ii  the  niilil.y  of  t  he  slaU'  and  llie 
wealth  of  the  agent.  However,  it  is  important  to  keep  in  mind  that  a  probabilistic  planning  method 
must  be  able  to  cope  with  cyclic  probabilistic  <lecision  graphs  as  w<'ll.  sinc<'  for  matty  planning 
problems  —  either  the  agent  cannot  avoid  to  repeat  some  of  the  stal.i's  or  it.  is  not  opt  imal  to  <lo  so. 
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as  shown  in  Figure  10,  and  then  use  folding-back  on  the  resulting  tree,  because  in 
general  «(ci  -f  Cj)  ^  «(ci)  -|-  ti(c2)  for  two  rewards  c\  and  C2  (that  is,  the  value  function 
is  no  longer  time-additive).  In  fact,  dynamic  programming  methods  cannot  be  used 
any  longer  in  any  way  without  considering  the  wealth  of  the  agent,  because  the  Markov 
property  does  not  necessarily  hold  for  risk-sensitive  agents  [Railfa.  IfKiS]. 

Consider  again  the  planning  problem  that  was  staU'd  in  Figure  (i.  I’Ih’  shan*d  subl  rcn* 
represents  the  choice  between  a  (h’terministic  lottery  A  (reward  -0.  IS  for  sure)  aiul 
a  non-deterministic  lottery  B  (rewards  -0.10  and  -1.00  with  ej|ual  probability).  As 
demonstrated  in  Figure  0  for  ».(c)  =  —  v/^,  tin*  agj'iit  should  choose  lol  l*'rv  B  if  it  has 
wealth  -0.10  when  deciding  Ix’twmi  tin’  two  lotteries.  IU>wevi‘r.  if  its  wealth  is  -1.00, 
it  shoidd  prefer  lottery  A.  'Phis  can  Iw  explained  as  follows;  The  agent  is  risk-sn'king. 
since  its  utility  function  —  v/^  is  convex,  but  the  convexity  flecr('as<*s  the  mor<*  negative 
c  gets.'*  Remember  that  the  wealth  of  the  agrud,  has  to  be  axhied  to  all  rewanls  »»f  a 

"For  a  qiiantirication  of  this  statement,  se*?  for  e.xaiTi|>le  [Pratt.  IJMH]. 
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lottery.  For  example,  if  the  agent  has  acquired  wealth  -0.10,  then  lottery  B  becomes 
“rewards  -0.20  and  -1.10  with  equal  probability.”  If  the  agent  has  already  accumulated 
cost  -1.0,  then  lottery  B  becomes  “rewards  -1,10  and  -2.00  with  equal  probability.” 
Thus,  the  more  negative  the  wealth  of  an  agent  is,  the  more  negative  the  total  rewards 
become  ajid  the  less  risk-seeking  the  agent  is.  Since  the  agent  accumulates  more  and 
more  action  costs  over  time,  it  becomes  less  and  less  risk-seeking. 

Since  the  optimal  action  for  a  state  depends  on  the  wealth  of  the  agent,  the  Markov 
property  does  not  hold  and  dynamic  programming  methods  cannot  be  n.sed  on  the 
original  probabilistic  decision  graph  without  taking  the  “wealth”  of  the  agent  into 
account,  which  makes  planning  very  inefficient.  This  problem  exists  only  for  risk- 
sensitive  planning,  but  not  for  risk-neutral  planning. 

One  can  circumvent  this  problem  with  planning  methotls  that  have  a  limited  look¬ 
ahead,  say  n.  The  planner  of  [Kanazawa  and  Dean,  1989],  for  example,  determines 
the  plan  that  generates  the  largest  expected  total  utility  in  the  first  ii  steps,  executes 
the  first  action  of  the  plan,  and  repeats.  Such  planning  methods  still  duplicate  shared 
subtrees  (since  they  “unroll”  the  underlying  Markov  decision  problem),  but  one  can 
now  control  the  amount  of  work  required  for  one  iteration  by  varying  t  he  look-ahead 
n.  Since  these  and  related  heuristic  planning  methods  rely  on  hill-climbing,  they  suffer 
from  the  limited  horizon  problem  and  their  success  depends  critically  on  the  structure 
of  the  planning  task. 


5  A  Solution 

In  the  following,  we  will  outline  one  possible  way  for  incorporating  risk-sensitive  at¬ 
titudes  into  arbitrary  planning  methods.  This  is  done  by  transforming  planning  ta,sk 
PT2  into  a  planning  task  PT.3,  which  can  then  be  solved  with  any  standard  (that  is. 
risk-neutral)  planning  method.  The  resulting,  optimal  plan  for  planning  t»usk  PT3  is 
optimal  for  the  risk-sensitive  planning  task  PT2  as  well.  The  key  to  accoTnplishing  this 
task  is  to  utilize  utility  functions  that  maintain  the  Markov  property. 

Consider  utility  functions  with  the  following  property  (called  "constant  local  risk  aver¬ 
sion”  [Pratt.  1964]  or  “delta  property”  [||owar<l  aiul  Matlu'son.  1972]):  if  all  rewards 
of  an  arbitrary  lottery  are  increased  by  an  arbitrary  amount,  tln'ii  the  <ertainty  e(|niv- 
alent  of  the  lottery  is  increased  by  this  amount  as  well.  The  only  utility  functions 
with  this  property  are  the  identity  function.  *onvex  «'xponential  functions  »(c)  = 
for  7  >  1.  concave  exponential  functions  (t(r)  =  -7'  for  0  <  7  <  I.  and  t  lu'ir  strictly 
positively  linear  transformations  [Watson  ami  Bnecle.  1987].  Sime  tlu'se  utility  fnm  - 
tions  are  parameterized  with  a  parameter  7,  oim'  can  exprc'ss  a  whoh-  spectrum  of 
risk-sensitivity  ranging  from  being  strongly  risk-aversf'  over  being  risk-neutral  to  being 
strongly  risk-seeking.  I'he  larger  7,  the  tnore  risk-s(‘eking  t  in'  agent  is.  and  vice  v«>rsa. 

'-‘‘Constant,  local  risk  aversion”  moans  that  the  quantity  ^7^,  which  isc.-ilhsi  "local  risk  .ivorsion." 
does  not  depend  on  the  reward  c. 
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These  utility  functions  are  exactly  the  ones  that  maintain  the  Markov  property  [Howard 
and  Matheson,  1972];  if  the  agent  executes  an  action  in  its  current  state  and  behaves 
optimally  afterwards,  then  it  faces  a  lottery.  There  is  one  lottery  for  every  action  that 
the  agent  can  execute  in  its  current  state.  The  lottery  with  the  largest  expected  utility 
or,  equivalently,  the  largest  certainty  equivalent  identifies  the  optimal  action.  Before 
determining  the  certainty  equivalents,  however,  one  has  to  add  the  wealth  of  the  agent 
to  all  (goal)  rewards  of  every  lottery.  This  increases  the  certainty  equivalent  of  every 
lottery  by  the  wealth  of  the  agent,  since  the  utility  function  has  the  delta  property. 
Thus,  when  comparing  lotteries,  one  can  ignore  the  wealth  of  the  agent. 

Howard  and  Matheson  [Howard  and  Matheson,  1972]  apply  utility  functions  with  the 
delta  property  to  Markov  decision  problems  with  finite  and  infinite  time  horizons.  In 
the  later  case,  they  assume  a  non-goal  oriented  task,  and  every  state-action  mapping 
has  to  determine  an  irreducible  (that  is,  strongly  connected)  Markov  cliain.  .As  shown 
in  [Koenig  and  Simmons,  1994],  their  analysis  can  be  applied  to  non-goal  oriented 
planning  and  reinforcement  learning  tasks  (“behaving  in  the  world”)  if  the  agent  is 
risk-sensitive  towards  variations  of  the  reward  that  it  receives  per  action  execution. 
Unfortunately,  our  goal-oriented  planning  task  PT2  does  not  possess  the  properties 
required  by  Howard  and  Matheson,  and  thus  we  either  cannot  proceed  in  exactly  the 
same  way  or  have  to  confirm  that  their  theory  is  still  applicable. 

5.1  Planning  for  Risk-Seeking  Agents 

In  the  following,  we  will  temporarily  restrict  our  attention  to  risk-seeking  agents  with 
utility  function  u{c)  =  Y  (or  strictly  positively  linear  transformation  thereof)  for 
risk  parameter  7  >  I.*"* 

First,  we  would  like  to  point  out  that  every  planning  task  PT2  can  be  I  ransfornied  into 
an  equivalent  planning  task  that  has  only  one  goal  state,  which  has  goal  reward  zero. 
The  argument,  however,  is  different  from  the  one  for  the  risk-neiitral  case.  Planning 
task  PT2  is  equivalent  to  the  following  planning  ta.sk  PT2’  for  a  risk-seeking  agent  ; 

Introduce  a  new  goal  state  with  goal  reward  zero,  arid  repla<*e  (‘very 
(old)  goal  state  s  €  G  with  a  non-goal  state  in  which  only  one  action 

*^As.sume  that  a  lottery  with  e.xpectwl  reward  r  is  repeal'd  »  tinu's  and  let  r,-  denote  the  reward 
received  on  the  /th  trial.  'I'lie  law  of  large  nninhers  shows  that,  as  n  inrre,a,s('s,  c,  approaclu's 

nc  and  the  variance  of  n  approaciw’s  zero,  'riins,  if  an  agent  can  participati'  in  a  lotU-ry  a  large 
number  of  times,  it  can  evaluate  the  lottery  according  to  r  no  inatti'r  what  its  atl  ii  ndt'  towards  risk 
is  provided  that  il  is  ntily  interrsird  in  its  final  wraith.  'I’his  might  not.  be  the  r;\sr.  how('ver.  .\s  an 
example,  considiT  an  agent  that,  ran  rhoost>  iwery  month  one  lotti'ry  from  a  given  set  of  lotteries  that 
supply  it  with  food  for  the  month.  'I'lie  agent  cannot  stork  mmsi'd  food,  since  it  spoils  within  ‘hirty 
days.  .Snrh  an  .agent  is  certainly  interesteil  in  the  expectcal  amount,  of  food  provided  by  a  lot  tery. 
However,  insteail  of  paying  attention  to  the  variance  of  ^"_i  r,  it  will  probably  be  mon'  intc'rested  in 
the  variance  of  c,-,  which  (lex's  not  vary  with  ii. 

''’We  will  show  tlx;  di'rivations  for  «(c)  =  j' ■  I'hey  can  ('.asily  Ix'  I'xU'iided  t.o  utility  functions  of  tlx* 
form  M(r)  =  my'’  ■+■  n  (for  rn  >  0),  although  tlnw«’  functions  do  not  incn'ase  tlx'  powe-r  of  tlx'  planning 
mechanism,  since  utility  functions  are  deKned  only  np  to  .strictly  positively  liix'ar  transformations. 


can  be  executed,  that  leads  with  probability  one  and  action  cost  r[.s]  — 
max*<gG  ^[^1  ~  1  <  0  to  the  new  goal  state.  Otherwise,  the  planning  problem 
remains  unchanged. 

Planning  task  PT2’  satisfies  all  requirements  of  our  planning  framework.  It  is  equivalent 
to  planning  task  PT2  in  the  following  sense:  the  better  a  plan  is  for  planning  task  PT2, 
the  better  it  is  for  planning  task  PT2’  as  well,  and  vice  versa.  This  is  the  case,  because 
the  certainty  equivalent  of  every  plan  for  planning  task  PT2’  is  maXs^gc  »■[«']  +  1  lower 
than  the  one  of  the  same  plan  for  planning  task  PT2.  Note  that  this  argument  depends 
on  the  utility  function  having  the  delta  property  —  it  does  not  hold  for  arbitrary  utility 
functions. 

In  the  following  sections,  we  will  show  how  to  calculate  the  expected  total  utility  of  a 
given  plan.  Then,  we  will  transform  the  planning  problem  into  one  for  a  risk-neutral 
agent  and  show  how  to  solve  it.  Finally,  we  will  demonstrate  our  ideas  on  a  small 
blocks-world  example  and  a  path-planning  example. 


5.1.1  Calculating  the  Expected  Total  Utility  of  a  Plan 

Assume  that,  for  some  planning  problem,  a  plan  (that  is.  a  state-action  mapping)  is 
given  that  assigns  action  a[s|  to  non-goal  state  s.  The  expected  total  utility  of  this 
plan,  that  is,  the  expected  total  utility  u[so]  of  its  start  state  .sq,  can  recursively  be 
calculated  as  follows. 

The  (expected)  total  utility  of  a  goal  state  s  is  tt[^)  =  it(r[.s])  =  After  the 

agent  has  executed  action  a[s|  in  a  non-goal  state  .s,  it  incurs  action  cost 
and  is  in  successor  state  s'  with  probability  I,,  state  .s',  it  faces  a  lottery 

again.  This  lottery  has  expected  total  utility  h[s']  anti  certainty  equivalent  = 

log..^u(s'].  According  to  the  axioms  of  utility  theory,  the  lottery  can  be  replacetl  with 
its  certainty  equivalent.  Then,  the  agent  incurs  a  total  rewartl  of  c''bl[.s,.s']  -|-  »“'(jt[.s']) 
with  probability  p“bl[5,s'].  Thus,  the  expected  total  utility  of  .s  can  be  calc\dat(Hl  as 
follows:*® 


«[.s 


*'€.S 


E 

.i'€.S 

E 

.5'€.S 

E 


//'W  [.s.  .s']  7'''’'''  7 "  “ '  < 


'®This  corre.sponds  to  the  policy-f^valiialion  .stop  in  [llow.ard  ;ind  Mathosoii,  l!)7‘2]  with  tlio  "ct'rtaiii 
equivalent  gain”  i]  =  0. 
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«'€S\G 

s'^G 

In  matrix  notation,  we  get  the  closed-form  solution 


[«[«]]»€S\G 


(id  -  b“W[^,^l7^“‘''f*’*'^],..'6S\G)-‘ 

•(p“W(s,s']7^  ^  ^]seS\G,»'6G[7’^^'’^]»€G 


where  id  is  the  identity  matrix  of  size  j5  \  Gp.  (The  matrix  inverse  in  the  expression 
always  exists.) 

For  7  approaching  one,  the  certainty  equivalent  of  any  state  for  any  plan  approaches 
the  expected  total  reward  of  that  state.  Therefore,  the  optimal  plan  for  a  risk-neutral 
agent  is  the  one  that  is  best  for  a  risk-seeking  agent  if  7  approaches  one. 


Proof:  Assume  that  the  execution  of  the  plan  leads  with  probability  p,  to 
total  reward  c,  if  the  agent  is  started  in  state  s.  Then,  the  expected  total 
reward  of  s  equals  HjPjC,  and  its  certainty  equivalent  is  «"‘(ZiP«w(Ct)). 

Thus,  lim^-,i«*‘(EiP.«(c,))  =  linH-i  log.,(i:,  p,7"-)  =  lim^-.i 

—  im/y_l  m-K— 1  p.^'i 


In  contrast,  for  7  approaching  infinity,  the  certainty  equivalent  of  any  state  for  any  plan 
approaches  the  total  reward  of  that  state  if  nature  acts  like  a  friend  (that  is,  chooses  the 
action  outcomes  not  with  a  coin  flip,  but  deliberately  so  that  it  is  best  for  the  agent).  Of 
course,  in  reality,  nature  does  flip  coins.  We  call  an  agent  that  assumes  (wrongly)  that 
nature  helps  it  as  much  as  it  can  and  calculates  it  total  utilities  accordingly  "‘extremely 
risk-seeking.”  Thus,  max-max-ing  (“both  the  agent  an<l  nature  maximize  the  total 
reward  for  the  agent”),  which  calculates  the  total  utility  of  a  non-goal  state  .s  for  a 
given  plan  as  m[.s]  =  max,,/g,s(c'*ld|..,^.s']  -|-?£(.s']),  determines  the  plan  that  is  lM*st  for  a 
risk-seeking  agent  if  7  approaches  infinity. 


Proof:  Assume  again  that  the  ex<H-iition  of  tiu'  plan  leads  wil  h  prob¬ 
ability  Pi  to  total  reward  r,  if  tin*  agent  is  started  in  stale*  .s.  I'lien. 
max.  r,  =  log.^  7'""’'' =  log^  />,7"''**"  ’  >  log.,  /»i7"  >  h>H-,(/'-'>' ' ) 

=  max, ( log.,  p,  -I-  Cj).  It  follows  that  max,r,  =  lim.v_,v.  max, f,  > 
lim.y_oc  log.v5ZiP«7"  ^  liriiY-.oo  n'ax,(log.,p,  -I-  fv)  =  inax.  r,,  and  tints 
linb,_oo  «^"'(ZiPi«(f:i))  =  iinW-.ooU)g.,EiPi7''  =  ntaxjCi. 
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5.1.2  Transforming  the  Planning  Problem 


To  show  how  every  planning  task  PT2  for  a  risk-seeking  agent  can  be  transformed 
into  an  equivalent  planning  task  PT3  for  a  risk-neutral  agent,  we  assume  again  that  a 
state-action  mapping  is  given.  We  use  the  same  symbols  for  planning  task  PT3  that 
we  used  for  PT2,  but  overline  them. 

Since  (without  loss  of  generality)  a  risk-neutral  agent  has  utility  function  u{c)  =  c,  it 
holds  that  ti[s]  =  t;[s].  A  goal  state  s  has  (expected)  total  utility  i2[.s]  =  ti(f[.s])  = 

The  expected  total  utility  of  a  non-goal  state  s  is 


a'es 

s'es 

Comparing  these  results  with  the  ones  in  the  previous  section  shows  that  h[.s]  =  m.[s] 
for  all  states  s  £  S  and  all  planning  problems  if  and  only  if  three  ecpialities  hold: 
f[s]  =  for  s  €  G.  Furthermore,  p“l*^[s,.s']  =  p“W[.s, and  c'M[.s..s']  =  0  for 
s  €  5  \  G'  and  s'  €  S. 

Thus,  planning  task  PT2  for  a  risk-seeking  agent  with  utility  function  «(c)  =  Y  's 
equivalent  to  the  following  planning  task  PT3  for  a  risk- neutral  agent: 

Introduce  one  additional  state  s  with  expected  total  reward  zero.  (State 
s  could  for  example  be  modeled  as  a  “sink”:  a  non-goal  state  in  which  only 
one  action  can  be  executed.  This  action  has  action  cost  zero  and  leads  l>ack 
to  state  s  with  probability  one.  State  .s  could  also  be  modeled  as  a  goal 
state  with  goal  reward  zero.)  Otherwise,  the  state  space,  action  s])a.ce.  start 
state,  and  goal  states  remain  unchanged.  The  goal  reward  of  any  goal  state 
s  ^  .s  is  7’'1®1.  When  the  agent  executes  action  n  in  a  non-goal  state  .s  .s. 
it  incurs  an  action  cost  of  zero  and  is  in  successor  state  s'  wit  h  t  ransit  ion 
probability  p'‘t''l[s,s']7‘'“***l’’*’l.  These  probabilities  do  not  sum  up  to  one. 

With  the  complementary  transition  probability  1  — 

the  agent  incurs  an  action  cost  of  zero  and  is  in  successor  state  .•<. 

Thus,  given  7,  one  transforms  planning  task  PT2  into  the  al)ove  planning  task,  for 
which  one  then  determines  the  plan  with  tlx*  largest  expectx'd  total  reward.  The  t  rans¬ 
formation  is  trivial  ainl  can  be  <lone  in  linear  time,  since  bot  h  r('pres('ntations  an'  of 
the  same  size. 

The  only  reason  lor  introducing  state  .s  is  to  make  the  probabilith's  sum  up  to  oiu'. 
Since  its  expected  total  reward  is  zero,  it  will  not  show  up  in  the  calcidations.  The 
specification  of  PT3  for  the  risk-seeking  planning  probhun  from  Figtiix'  (i  is  shown  in 
Figure  11.  (State  .s  is  modeled  as  a  non-goal  state.)  Nol.e  that,  although  t.lnw  can  bot  h 
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Figure  11:  Transformation  11  (does  work) 


be  expressed  with  probabilistic  decision  graphs  of  the  same  topology,  the  specification 
of  the  planning  problem  for  PT3  differs  fundamentally  from  the  one  of  PTl.  For 
example,  an  obvious  difference  is  that  all  actions  of  planning  task  PT3  have  action  cost 
zero.  Therefore,  action  costs  can  be  ignored  for  risk-sensitive  planning. 

5.1.3  Finding  Optimal  Plans 

Planning  task  PT3  can  be  solved  with  probabilistic  AI  planning  methods  or,  alterna¬ 
tively,  with  dynamic  programming  methods,  called  Markov  decision  algoritliins. 

It  is  interesting  to  note  that  the  plan  with  the  largest  expected  total  utility  is  not 
necessarily  admissible  even  if  an  admissible  plan  exists,  as  shown  in  Figure  12  for  a 
risk-seeking  agent  with  iitility  function  h(c)  =  2'  .  If  the  agi'iit  «  hoos<'s  action  .\.  llu'ii 
the  expec.terl  total  utility  of  the  plan  is  O.oO/if  — x)) -|-0.r)0n(  — | )  =  0.2r).  but  tiu'  plan  is 
not  admissible.  If  the  agent  chooses  ;u:tion  II,  then  it  achieves  a.  U)tal  (<\\p<'cU'*|)  utility 
of  l.00«(— 3)  =  0.125  and  reaches  the  goal  state  for  sure.  'Phus,  tlu*  inadmissibh'  plan 
results  in  a  larger  <*xpec,te<l  total  utility.  'Phis  cannot  happen)  for  risk-n<*ntral  agenits, 
since  the  optimal  risk-neutral  plan  is  always  admissible  (if  an  admissible  plan  e’xists). 


The  reason  why  the  plan  with  the  largest  expected  total  utility  is  tio  longer  guarante<Hl 
to  be  admissible  for  risk-seeking  agents  is  shown  in  Figure  13:  While  for  risk-neutral 
agents  all  inadmissible  plans  have  certainty  equivalent  minus  infinity,  this  is  no  longer 
true  for  risk-seeking  agents.  The  table  also  shows  that  this  “problem"’  cannot  arise  for 
risk-averse  agents. 

It  should  be  pointed  out  that,  from  the  standpoint  of  utility  theory,  there  is  absolutely 
no  problem  with  optimal  plans  that  are  inadmissible.  In  the  AAAI  robot  conifxMition 
example,  for  instance,  one  might  risk  that  the  robot  will  not  be  able  to  finish  the 
competition  with  a  certain  probability.  However,  if  one  insists  on  using  utility  tlu*ory 
only  to  choo.se  the  best  plan  among  all  admissible  plans,  one  can  utilize  that  the  optimal 
plan  for  a  risk-.seeking  agent  is  guaranU^I  to  be  a<lmissible  if  rrrry  sl.at<’  is  solvable. 
Thus,  if  some  states  are  iinsolvable,  one  can  easily  remove  t  he  unsolvable  states  from 
the  planning  problem  before  solving  it  [Ko<niig,  1?)!)!].  'I'he  optimal  plan  of  t.lu'  result  ing 
planning  problem  is  then  guarantwfl  to  be  admissibh*. 

The  optimal  plan  for  the  transformer!  planning  task  I*'!';!  r  an  lx*  d<’t(<rmin<'d  with 

'^Properties  such  as  this  one  can  easily  be  provetl  using  tlx*  any-tiiix*  pro|>erly  of  Markov  derision 
algorithms  such  as  the  policy-iteration  algorithm. 
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dynamic  programming  algorithms.  If  the  transformed  probabilistic  decision  graph  is 
acyclic,  it  can  be  solved  in  linear  time  with  folding-back.  For  cyclic  probabilistic 
decision  graphs,  it  can  be  formulated  as  Markov  decision  problem,  that  can  then  be 
solved  in  polynomial  time  with  Markov  decision  algorithms.  The  representation  as 
Markov  decision  problem  proves  that  one  can  indeed  restrict  plans  to  state-action 
mappings  without  losing  optimality  and,  furthermore,  that  there  exists  at  least  one 
state-action  mapping  that  is  optimal  for  all  possible  start  states.'”  It  turns  out  that 
the  Markov  decision  problems  for  planning  task  PT3  have  a  simpler  structure  than  the 
ones  for  PTl  (namely,  all  state-action  mappings  determine  absorbing  Markov  chains). 
This  simplifies  the  optimization  algorithms. 

In  order  to  determine  an  optimal  plan  for  planning  task  PT3,  one  can  for  example  use 
value-iteration  [Bellman,  1957],  policy-iteration  [Howard,  1964],  Q-Iearning  [Watkins, 
1989],  or  linear  programming.  As  an  example  of  such  a  dynamic  programming  tech¬ 
nique  consider  (a  simplified  version  of)  Howard’s  (single-chain)  policy-iteration  algo¬ 
rithm  [Howard,  1964]  [Howard  and  Matheson,  1972].  One  can  either  use  the  algorithm 
on  the  transformed  planning  task  PT3  or,  as  we  have  done  here,  adapt  the  algorithm 
so  that  it  works  on  the  original  planning  task  PT2: 


1.  Choose  an  arbitrary  state-action  mapping  «[.?]  G  .4(s)  for  all  .s  €  S  \  G. 

2.  (value-determination  operation)  Set 


[«W]*€S\G 


3.  If  no  «[s]  for  any  s  G  S'\  G  has  changed  in  the  previous  step  (from  the  value  that 
it  had  in  the  previous  iteration),  then  stop.  An  optimal  state-action  jnapping  is 
to  select  action  a[s]  in  state  s  G  S’  \  G. 

4.  (policy-improvement  routine)  Set  for  every  s  G  S  \  G 

a[.s]  ;=  argmax„g^,,)(  Y.  p''[-s,.s']7‘'“'’'*'’ »[•■<'] 

veG 


5.  Go  to  2. 

This  algorithm  is  an  any-time  algorithm.  'The  term  ‘‘aiiy-tinu'  algorilhm"  was  coiinMl 
by  [Bofidy  atid  Dean,  l9(Sf)]),  aiul  [Bresina  and  Drummond.  1990]  first  develop«'«|  an 
any-time  planner.  Any-1.im<'  planning  inethoils  ran  l)t'  >ts«'d  to  d<'t('rmiiH'  a<  <'or«ling 
to  decision-tlu'oretic  criteria  —  when  to  stop  planning  and  start  ('X<'cuting  the  plan, 
becau.se  the  possibh'  future  increase  in  plan  <|uality  does  not  justify  the  effort  of  planning 

'‘See  [Oortsckas,  1987]  for  a  gocxi  rcvifw  of  Markov  derision  tln'ory. 
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any  further.  The  policy-iteration  algorithm  is  an  any-time  algorithm  in  the  sense  that 
the  expected  total  utility  of  no  state  can  decrease  from  one  iteration  to  the  next,  but 
the  expected  total  utility  of  at  least  one  state  strictly  increases,  until  the  optimal  state- 
action  mapping  is  found  in  finite  time  [Howard,  1964].  Thus,  the  expected  total  utility 
of  the  currently  best  plan  cannot  decrease  from  iteration  to  iteration.  A  solved  state 
remains  solved  in  the  following  iterations  and  an  admissible  plan  stays  admissible. 

When  implementing  a  risk-sensitive  planning  method,  however,  one  will  usually  only 
want  to  approximate  optimal  plans  and  therefore  use  more  incremental  (and  faster) 
dynamic  programming  techniques  that  restrict  their  attention  to  certain  states  without 
having  to  look  at  the  whole  state  space,  determine  for  which  states  accurate  expected 
total  utilities  are  not  that  important,  and  utilize  initial  heuristic  knowledge  (see  [Dean 
et  al.,  1993]  and  [Moore  and  Atkeson,  1993b]  for  approaches  in  this  direction).  Dynamic 
programming  methods  for  risk-sensitive  planning  can  also  be  extended  to  cases  where 
the  action  models  are  not  given  in  advance,  but  have  to  be  learned  from  reinforcement 
that  is  provided  externally  in  response  to  action  executions  (“experiments")  of  the 
agent  (so  called  “reinforcement  learning”). 

However,  dynamic  programming  algorithms  are  brute-force  search  algorithms,  since 
they  do  not  utilize  available  domain  knowledge  such  as  how  different  actions  interact 
with  each  other.  AI  planning  methotls,  on  the  other  hand,  are  knowledge-based.  .Al¬ 
though  AI  planning  methods  are  usually  not  able  to  guarantee  the  optimality  of  their 
plans,  they  can  be  used  for  risk-seeking  planning  instead  of  Markov  decision  algorithms. 
The  larger  the  expected  total  reward  of  the  plan  that  they  determine  for  planning  task 
PT3,  the  larger  is  the  expected  total  utility  of  the  same  plan  for  the  corresponding 
planning  task  PT2. 

Instead  of  planning  methods  that  maximize  or  satisfice  expected  total  reward,  we 
can  also  use  AI  planning  methods  that  maximize  or  satisfice  the  probability  of  goal 
achievement,  but  do  not  take  cost  considerations  into  account.  First,  we  have  to 
transform  planning  task  PT2  into  the  equivalent  planning  task  PT2’.  Remember  that 
PT2’  is  a  risk-seeking  planning  task  that  has  only  one  goal  state,  which  has  goal 
reward  zero.  Now  consider  the  risk-neutral  planning  task  PT3  that  corresponds  to 
planning  tcisk  PT2’  and  assume  that  state  .s  has  b<*en  modeled  as  a  non-goal  state. 
Since  the  total  utility  of  the  goal  state  of  PT2’  is  7“  =  1.  it  has  goal  reward  one  for 
planning  task  PT3.  Because  all  actions  of  planning  task  PT3  have  act  ion  cost  z('ro. 
the  probability  of  reaching  the  goal  state  of  PT3  for  any  plan  e(|nals  its  j'xpc'cteil 
total  reward  for  the  same  planning  task,  which  in  turn  etpials  its  expected  total  utility 
for  planning  task  PT2.  Thus,  planning  methods  that  determine  i)lans  that  maximize 
the  probability  goal  achievement  (or  whose  probability  of  goal  achievt'imMit.  ('xce«‘«ls  a 
given  probability)  can  be  \ise<l  t,o  maximize  (or  satisfice)  ('xpected  total  utility  (or.  in 
the  limit,  exp<*ctefl  total  rewaixl).  Although  one  cannot  apply  tlu'se  planiiing  nuM  hods 
directly,  otu'  can  first  transform  the  planning  probh’m  and  tluMi  apply  tlu'  planning 
methods  to  the  transforriK'fl  planning  prol>lem.  I'hns.  pt'ihaps  surprisingly,  they  rnit 
be  used  to  take  cost  consitlerations  into  account  providecl  that  they  an*  abh'  to  deal 
with  planning  tasks  for  which  not  all  states  are  solvabh'.  In  fact.,  ev<'ry  non-goal  stat.<'  of 
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agent  is... 

certainly  equivalent  n*' (ir  [.(„] )  of... 
an  admissibie  plan  an  inadmissible  plan 

lisk-neuual 

{«  Uol )  >  ("~)  “  '  (“  I'llI  )  • 

risk-seeking 

risk-averse 

a"'  (u  (.t„) )  2  —  «*'  (« [.r„) )  *  -<» 

(utility  function  has  della  property) 

Figure  13:  Possible  Certainty  Equivalents  of  Plans 

PT3  is  unsolvable  (even  if  every  non-goal  state  of  PTl  is  solvable),  since  the  unsolvable 
state  3  can  be  reached  from  every  non-goal  state  with  positive  probability.  Therefore, 
finding  optimal  plans  is  not  solely  a  matter  of  extending  the  number  of  states  that  are 
covered  by  the  state-action  mapping  (the  so-called  “plan  envelope”),  since  the  agent 
cannot  recover  from  state  s  and  thus  has  to  minimize  the  probability  of  reaching  this 
state  if  it  wants  to  determine  an  optimal  plan. 

To  summarize,  planning  task  PT3  can  be  solved  with  AI  planning  methods  that  either 
maximize  (or  satisfice)  the  expected  total  reward  or,  equivalently,  the  probability  of 
goal  achievement.  Whether  state  5  should  be  modeled  as  a  goal  state  or  non-goal  state 
depends  on  which  one  of  the  two  kinds  of  planning  methods  one  wants  to  use.  If  s  is 
modeled  as  a  non-goal  state,  then  —  as  mentioned  in  the  previous  paragraph  -  no  plan 
is  admissible  for  PT3  if  the  start  state  is  a  non-goal  state.  In  this  case,  it  is  sensible 
to  rank  plans  according  to  their  probability  of  goal  achievement.  If  .s  is  modeled  a.s  a 
goal  state,  then  all  plans  are  admissible  for  PT3  and  it  makes  sense  to  choose  among 
them  according  to  their  expected  total  reward. 

5.1.4  Example  1:  A  Stochastic  Blocks- World 

We  use  the  blocks-world  problem  that  is  stated  in  Figure  14  to  illustrate  our  ideas.  Its 
state  space  contains  162  states.  Figure  15  shows  four  of  the  state-action  mappings  t  hat 
solve  it,  and  Figure  16  illustrates  how  the  certainty  equivalents  l|■*(f/.[.so|)  =  log.^  h[so] 
of  the  four  plans  vary  with  the  natural  logarithm  of  the  risk  parameter  7  (calculat'd 
for  example  with  the  policy- iteration  algorithm  stated  above,  which  —  depending  on 
the  initial  policy  —  typically  converges  after  aroiiml  live  iterations). 

Plan  A,  that  involves  no  risk  atui  can  be  ex<'cutefl  in  six  minnt's  (that  is.  has  total 
reward  -6.00),  has  the  largest  expected  total  ix'wartl  of  all  plans  (iiol,  just.  I, he  four  plans 
shown)  and  will  therefore  be  chosen  by  a  risk-neutral  agent.  IIow<‘Vf'r.  plan  A  is  not 
necessarily  optimal  for  a  risk-se^eking  agent.  When  using  plan  D,  for  example,  l.ln'  ag<'nl. 
reaches  a  goal  state  in  only  thret;  minutes  if  it  is  lucky. 
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goal 


There  ate  seven  goal  states,  all  of  which  are  equally  preferable. 

In  every  blocks-world  state,  one  can  move  a  block  that  has  a  clear  top 
onto  either  the  table  or  a  different  block  that  has  a  clear  top,  or  paint  a 
Mock  white  or  black. 

Moving  a  block  takes  only  one  minute  to  execute,  but  is  very  unreliable. 
With  probability  0.10,  the  moved  block  ends  up  at  its  intended  destina¬ 
tion.  With  probability  0.90,  however,  the  gripper  loses  the  block  and  it 
ends  up  directly  on  the  table.  (Thus,  moving  a  block  to  the  table  always 
succeeds.) 

Painting  a  block  takes  three  minutes  and  is  always  successful. 


Figure  14:  A  Blocks- World  Problem 


The  optimal  plan  for  a  risk-seeking  agent  is  the  one  with  the  largest  e.xpected  total 
utility  or,  equivalently,  certainty  equivalent.  Since  Plan  A  is  deterministic,  its  certainty 
equivalent  equals  the  (expected)  total  reward  of  its  start  state,  no  matter  what  the 
risk-attitude  of  the  agent  is.  The  other  three  plans  are  non-deterministic.  Thus,  their 
certainty  equivalents  increase,  the  more  risk-seeking  the  agent  becomes,  and  different 
plans  can  be  optimal  for  different  degrees  of  risk-seeking  attitude.  Figure  16  shows 
that  plan  A  is  optimal  in  the  interval  In 7  €  (0.00,0,93].  For  In 7  €  [0.94, 4. .58],  plan  C 
is  optimal,  and  plan  D  should  be  chosen  for  In 7  €  [4..59.  oc).  (These  statements  hold 
for  all  plans,  not  just  the  four  plans  shown  in  the  figure.) 

For  In 7  approaching  zero  (that  is,  7  approaching  one),  the  certainty  equivalent  of  any 
plan  approaches  its  expected  total  reward.  For  plan  D,  for  example,  the  certainty 
equivalent  approaches  -21.00.  (See  Figure  21  to  verify  this.)  Since  plan  A  is  opti¬ 
mal  for  In  7  approaching  zero,  it  is  optimal  for  a  risk-neutral  agent.  In  contrast,  for 
In 7  approaching  infinity  (which  is  equivalent  to  7  approaching  infinity),  the  certainly 
equivalent  of  any  plan  approaches  its  total  reward  for  the  case  that  nature  acts  as  a 
friend.  When  executing  plan  D,  for  example,  the  agent  can  reach  a  goal  state  in  only 
three  minutes  if  it  is  lucky.  Thus,  the  certainty  ecjuivalent  approaches  -3.00,  aiul  plan 
D  is  optimal  for  an  extremely  risk-seeking  agent.  (The  certainty  etpiivaleiit  of  plan 
C  also  approaches  -3.00.  Thus,  it  is  also  optimal  for  an  extremely  risk-scx'king  agent. 
However,  for  In7  >  4  ..')9  plan  D  <lominates  plan  ('  for  a  risk-s('eking  agent  in  that  it 
always  has  a  larger  certainty  ec|uiva!ent.) 

In  order  to  be  able  to  apply  probabilistic  planning  methods  oth(*r  than  .Vlarkov  dc'cision 
algorithms,  we  explicitly  transform  the  planning  problem  into  one  for  a.  risk-nentral 
agent.  The  original  planning  problem  can  for  example  b«’  expr<‘sse<l  with  augnx’ntetl 
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Certainty  Equivalents  of  some  Plans  for  Different  Degrees  of  Risk-Seeking  Attitude  gamma 


Figure  16:  Certainty  Equivalents  of  the  Four  Blocks- World  Plans  (Risk-Seeking  Agent) 


STRIPS-rules**  [Koenig,  1991],  three  for  the  move  actions  (“move  block  X  from  the 
top  of  block  Y  on  top  of  block  Z,”  “stack  block  X  on  top  of  block  Y,"  “unstack  block 
X  from  block  Y”)  and  one  for  the  paint  action  (“paint  block  X  with  color  C”).  The 
first  move  action  can  be  expressed  as  follows: 


move(X,Y,Z) 
precond : 

outcome : 
prob: 
reward : 
delete : 
add: 

outcome : 
prob: 
reward : 
delete : 
add: 


on(X,Y),  clear(X),  clear(Z),  block (X) ,  block(Y) , 
block(Z) ,  unequal(X,Z) 

/♦  the  primary  outcome  ♦/ 

0.1 

-1 

on(X,Y),  clear(Z) 
on(X,Z),  clear(Y) 

/*  failure:  block  X  falls  onto  the  table  */ 

0.9 

-1 

on(X,Y) 

clear(Y) ,on(X, table) 


‘^The  original  STRIPS-notation  [Fikes  and  Nilsson,  1971]  applies  t.o  deterministic  domains  only. 
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The  transformation  changes  the  transition  probabilities,  action  costs,  and  goal  rewards. 
In  accordance  with  our  earlier  comments,  we  do  not  model  the  transition  into  state  s. 
The  goal  states  remain  goal  states,  but  now  get  assigned  a  goal  reward  of  one.  The 
STRIPS-rules  are  transformed  as  shown  in  Section  5.1.2.  For  example,  for  7  =  2,  the 
above  STRIPS-rule  is  transformed  into  the  following  one: 


move(X,Y,Z) 
precond : 

outcome: 
prob: 
reward : 
delete: 
add: 

outcome : 
prob: 
reward : 
delete : 
add: 


on(X,Y),  clear(X),  clear(Z),  block(X) ,  block(Y), 
block(Z),  unequal(X,Z) 

/*  the  primary  outcome  */ 

0.05 

0 

on(X,Y),  clear (Z) 
on(X,Z),  clear(Y) 

/*  failure:  block  X  falls  onto  the  table  */ 

0.45 

0 

on(X,Y) 

clear (Y) ,on(X, table) 


With  the  complemerrtary  probability  (0.5),  the  action  execution  results  in  the  new 
state  5,  that  has  a  total  reward  of  zero.  Now,  one  can  use  any  planning  method  that 
maximizes  expected  total  reward  or,  in  this  case  equivalently,  the  probability  of  goal 
achievement  on  the  transformed  STRIPS-rules  to  determine  an  optimal  plan  for  the 
risk-seeking  agent.  (If  the  probabilistic  planning  method  maximizes  expected  total 
reward,  it  might  be  advantageous  to  model  state  as  a  goal  state  with  goal  reward 
zero.  If  the  planner  maximizes  the  probability  of  goal  achievement,  state  .s  has  to  be 
modeled  as  a  non-goal  state  from  which  no  goal  state  can  ever  be  reached.) 


5.1.5  Example  2:  Stochastic  Path-Planning 

As  a  second  example,  consider  the  simple  path-planning  domain  shown  in  Figure  17. 
The  states  of  this  grid-world  correspond  to  locations.  In  every  state,  the  agent  has  at 
most  four  actions  available,  namely  to  go  up,  right,  down,  or  left.  All  actions  take  the 
agent  one  minute  to  execute,  but  they  are  not  necessarily  deterministic.  They  succeed 
with  probability  but  their  outcome  deviates  ninety  degrees  to  the  left  or  right  of 
the  intended  direction  with  probability  each.  Thus,  x  €  [0, 1)  is  a  parameter  for 
the  certainty  of  the  actions:  the  larger  the  value  of  .r  is,  the  more  certain  ar<'  their 
outcomes.  Actions  have  deterministic  outcomes  if  .r  =  I;  their  intend<'d  outcoiiH*  and 
its  two  deviations  are  equally  likely  for  the  other  extreme,  .r  =  0. 

In  every  state,  the  agent  can  execute  all  of  the  actions  whose  intended  dinn-tion  is  not 
immediately  blocked  by  a  wall.  Besides  standard  walls,  the  grid-world  also  (’ontains 
‘‘one-way  walls,''  that  can  be  traversed  from  left  to  right,  but  not  in  the  op|>osite 


I  wall 

L  one-way  wall 

(can  be  ireveneil  fram  left  lo  right) 

.naie  bonier 

Figure  17:  A  Path-Planning  Problem 
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direction.  (They  might  for  example  be  steep  slopes  that  the  agent  can  slide  down,  but 
not  climb  up.)  If  the  agent  executes  an  action  and  it  has  the  intended  outcome,  the 
agent  cannot  bump  into  a  wall.  However,  running  into  a  wall  is  possible  for  unintended 
outcomes,  in  which  case  the  agent  does  not  change  its  location.  As  an  example,  consider 
state  X  in  figure  17  and  assume  that  x  =  0.5.  In  this  state,  the  agent  can  go  up,  right, 
or  down.  If  it  tries  to  go  up,  it  will  succeed  with  probability  0.6,  unintentionally  go  right 
with  probability  0.2,  and  unintentionally  remain  in  state  X  with  the  same  probability. 

The  agent  can  reach  the  goal  state  from  the  start  state  on  two  tlifferent  paths.  If  the 
actions  have  deterministic  effects  (that  is,  if  x  =  I),  the  traversal  of  path  B  takes  13 
minutes  and  the  one  of  path  A  15  minutes.  Thus,  the  agent  prefers  path  B  over  path 

A,  independently  of  its  risk-attitude.  If  the  actions  do  not  have  deterministic  effects, 
however,  the  agent  risks  to  traverse  a  one-way  wall  unintentionally  when  following  path 

B,  in  which  case  it  has  to  retrace  parts  of  its  path.  Since  path  B  is  better  than  path  A 
in  the  best  case,  we  expect  path  A  to  become  less  attractive  when  the  agent  Ijecomes 
more  risk-seeking. 

Figure  18  shows  how  the  value  of  the  action  certainty  parameter  .r  that  makes  the  agent 
indifferent  between  the  two  paths  varies  with  the  risk-attitude  of  the  agent  (expressed  as 
the  natural  logarithm  of  risk  parameter  7).  The  figure  actually  comprises  risk-seeking, 
risk-neutral,  and  risk-averse  behavior.  Here,  we  are  only  interested  in  the  risk-swking 
part:  ln7  >  0.  If,  for  a  given  risk-attitude,  the  actual  value  of  x  is  below  the  graph, 
then  the  agent  chooses  path  A,  otherwise  it  chooses  path  B.  As  expected,  the  value  of  x 
that  makes  the  agent  indifferent  between  the  two  paths  decreases  the  more  risk-seeking 
the  agent  becomes.  For  In  7  approaching  infinity,  it  approaches  zero;  an  extremely 
risk-seeking  agent  prefers  path  B  over  path  A,  since  path  B  can  be  traversed  in  13 
minutes  in  the  best  case,  whereas  path  A  cannot  be  traversed  in  less  than  15  minutes. 


5.2  Planning  for  Risk- A  verse  Agents 


For  risk-averse  agents,  one  can  proceed  as  outlined  for  risk-seeking  agents  in  the  pre¬ 
vious  section.  In  this  case,  one  has  to  use  a  function  from  the  famih'  «(r)  =  —7"' 
(or  any  strictly  positively  linear  transformation  thereof)  for  0  <  7  <  1.  Al¬ 
though  the  values  can  no  longer  be  interpreted  as  probabilities  (since 

>  1),  one  can  use  the  same  methods  as  in  the  risk-seeking  case 
if  one  takes  care  of  one  complication:  The  solution  ii[.so]  of  the  system  of  litiear  e<|ua- 
tions  from  Section  5.1.1  can  now  be  finite  even  for  plans  that  have  expected  total 
utility  minus  infinity.  The  planning  methods  can  t  hen  erroneously  return  such  plans 
as  optimal  solutions.  Fortunately,  these  plans  are  easy  to  characterize  (“plans  that 
have  at  least  one  cycle  with  ‘probability’  gr<‘a.t,er  than  one”),  and  one  <-au  remedy  the 
problem  by  either  initializing  the  <lynamic  programming  algorithms  more  r«*stri<-tedly 
or  extending  them  slightly.'*'* 


'  (Koenig  and  Simmons,  1994]  for  ^letails.  For  plans  t  hat,  have  cych's  with  “prol>al)ility”  greater 
than  or  equal  to  one.  the  dynamic  programming  equation  is  no  longer  a  contraction  mapping.  The 
case  of  cycles  with  “probability"  larger  than  one  can  be  approachinl  similarly  to  the  rase  of  cych's  with 
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Action  Certainty  Cor  which  the  Agent  is  Indifferent  between  the  Two  Paths 


Figure  18:  Action  Certainty  for  which  the  Agent  is  Indifferent  between  the  Two  Paths 


For  7  approaching  zero,  the  certainty  equivalent  of  any  state  for  any  plan  approaches 
the  expected  total  reward  of  that  state  if  nature  acts  as  an  enemy  (that  is.  chooses 
action  outcomes  not  with  a  coin  flip,  but  deliberately  so  that  it  is  worst  for  the  agent). 
We  call  an  agent  that  assumes  (wrongly)  that  nature  hurts  it  as  much  as  it  can  and 
calculates  its  total  utilities  accordingly  “extremely  risk-averse.”  (Such  an  agent  believes 
in  Murphy’s  law:  If  anything  can  go  wrong,  it  will.)  They  have  recently  been  studied 
in  the  AI  literature  by  [Heger,  1992]  and  [Moore  and  Atkeson.  199:la].  Thus,  max- 
min-ing  (“the  agent  maximizes  and  nature  minimizes  the  total  reward  of  the  agent.”  in 
game-theoretical  contexts  also  called  min-max-ing),  which  calculates  the  total  utility 
of  a  non-goal  state  s  for  a  given  plan  as  «[.*(]  =  min,*  -|-«[s']),  <letermines  the 

plan  that  is  best  for  a  risk-averse  agent  if  7  approac-hes  zero.  (l''or  7  approachitig  one. 
the  certainty  ef|uivalent  of  any  state  for  any  plan  approaches  —  as  in  t  in*  risk-se<'king 
case  —  the  expected  total  reward  of  that  state.) 

If  there  are  cycles  in  probal>ilistic  decision  graphs,  then  unrortiinat<'ly  ihe  «>x- 

probability  on«,  which  —  as  is  well  known  in  tin*  field  ot  Markov  decision  theory  can  also  l>e  solved  by 
either  initializing  the  dynamic  programming  algorithms  mon'  restrictedly  [Koenig  anti  Simmons,  nil)<l] 
or  extending  them  slightly,  see  Tor  example  the  multiple-chain  policy-iti'ration  algorithm  [Howard, 
196d). 
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Figure  19:  A  Plan  for  Stacking  Two  Blocks 


pected  total  utilities  of  admissible  plans  (and  thus  their  certainty  equivalents)  can  be 
minus  infinity,  see  also  Figure  13.  Imagine  for  example  an  extremely  risk-averse  agent. 
Thus,  given  a  plan,  the  agent  assumes  that  nature  will  try  to  keep  it  away  from  a 
goal  state.  The  agent  assigns  a  plan  an  expected  total  utility  of  minus  infinity  ('Mt 
is  scared”)  if  a  vicious  nature  could  indeed  prevent  it  from  reaching  a  goal  state.  In 
this  case,  utility  theory  might  no  longer  be  able  to  distinguish  admissible  plans  from 
inadmissible  ones.  Figure  14  shows  that  this  problem  can  not  arise  for  risk-neutral  or 
risk-seeking  agents. 

Consider  for  example  the  plan  shown  in  Figure  19  that  solves  the  problem  of  stacking 
two  blocks  with  a  move  action  that  fails  with  probability  p.  The  total  reward  of  state 
.So  is  —i  —  1  with  probability  p‘(l  —  p)  (for  all  integers  i  >  0).  Thus 


«N  = 

t=0 

tsO 


I  ^oo  otherwise 


How  the  certainty  e(|iiivalent  of  the  plan  depeiuls  on  p  is  sliown  in  l■’iglu•e  20.  'PIm' 
expected  total  utility  of  the  plan  is  minus  inlinity  for  p  >  7.  If  the  a.gent  <-an  choose 
between  a  move  action  with  action  failure  probability  pi  and  a  different  move'  action 
with  action  failure  probability  pj  where  pi  >  //-j  >  7,  it  cannot  elecirle  which  one*  to 
prefer  although  it  should  clearly  choose  the  later  move  action  over  the  former  one. 

The  example  also  demonstrates  that  plans  can  have  fixe<l  points  that  do  not.  corrc'spond 
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Figure  20:  Certainty  Equivalent  of  the  Plan  for  Stacking  Two  Blocks 


to  their  expected  total  utility.  The  following  system  of  linear  equations  is  the  one  used 
in  Section  5.1.1  to  calculate  the  expecterl  total  utilities  of  the  states: 


m[so]  =  P7~‘«M  +  { I  -  P)!”'”!**!] 
«[^i]  =  -7° 

Its  solutions  are 


«(.9o]  = 


l-p 


P-1 

«(s,]  =  - 1 


This  shows  that,  fot  p  >  7,  the  expectetl  total  utility  of  th<‘  plan  dilh'rs  from  tln'  value* 
of  u[.s,)]. 

Finally,  consider  again  the  example  domains  from  Swtions  5.1.1  ami  5.1.5.  Figure  21 
shows  how  the  certainty  equivalents  of  the  four  plans  for  the  hlocks-world  problem 
vary  with  the  natural  logarithm  of  the  risk  parameter  7  if  the  agent  is  risk-averse. 
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Certainty  Equivalents  of  some  Plans  for  Different  Degrees  of  Risk-Averse  Attitude  gamma 


Figure  21:  Certainty  Equivalents  of  the  Four  Blocks- World  Plans  (Risk- A  verse  Agent) 


The  optimal  plan  for  such  an  agent  is  always  plan  A,  independent  of  7.  Although  the 
certainty  equivalent  of  plan  A  is  defined  for  all  values  of  In  7,  the  certainty  equivalents 
of  plans  B,  C,  and  D  are  finite  only  for  —0.1 1  <  ln7  <  0  (that  is,  0.9  <  7  <  1 ).  They 
are  minus  infinity  for  smaller  values  of  In  7.  For  the  path-planning  domain.  FigJire  18 
already  contains  the  results  for  negative  attitudes  towards  risk,  that  is,  for  In  7  <  0. 


6  Conclusion 

In  this  paper,  we  were  concerned  with  probabilistic  planning  for  lisk-finmlirr  agents, 
since  there  are  many  situations  where  it  i.s  not  optimal  to  determine  plans  that  minimize 
expected  total  execution  cost  or  maximize  the  |)rol)ability  of  goal  aehiewment.  We  used 
acyclic  and  cyclic  probabilistic  decision  graphs  a.s  S'l'RIPS-like  planning  framework 
and  utilized  utility  functions  that  possess  the  delta  prop*  -ty.  Tin'S!’  utility  functions 
cover  a  whol«!  spectrum  of  risk-.sensitive  attituch's  from  being  strongly  risk-av<'rse  «iv<’r 
being  risk-neutral  to  being  strongly  risk-wvking.  'I'ln'y  fill  a  gap  I»’tw<'<'u  approaclu's 
previously  studierl  in  the  A I  literature,  namely  the  approach  to  minimize  exp«'cted  (.otal 
execution  cost  (risk-netitral  attitude)  and  the  approach  to  assume  that  uatuix'  acts  liki' 
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Figure  22:  Continuum  of  Risk-Sensitive  Behavior 


a  friend  (extremely  risk-seeking  attitude)  or  enemy  (extremely  risk-averse  attitude), 
all  of  which  they  can  asymptotically  approximate  as  shown  in  Figure  22.  We  obtained 
the  following  results: 

While  it  is  indeed  true  that  —  in  order  to  incorporate  risk-seeking  attitudes  into  plan¬ 
ning  methods  —  one  only  has  to  change  the  planning  problem  and  can  leave  the 
planning  method  untouched,  it  is  not  enough  to  replace  all  costs  and  rewards  with 
their  respective  utilities. 

For  acyclic  probabilistic  decision  graphs,  one  could  first  propagate  all  action  costs  to 
the  value  nodes,  replace  all  accumulated  costs  with  their  respective  utilities,  and  then 
use  folding-back  to  determine  the  optimal  decisions  for  the  decision  nodes.  This  works 
for  all  utility  functions,  although  one  might  be  forced  to  duplicate  all  shared  subtrees, 
which  makes  the  run-time  in  the  worst  case  exponential  in  the  size  of  the  probabilistic 
decision  graph.  However,  if  the  utility  functions  possess  the  delta  property,  one  can 
easily  transform  the  acyclic  probabilistic  decision  graph  into  a  different  probabilistic 
decision  graph  of  equal  size,  that  one  can  then  optimize  for  a  risk-neutral  agent  in 
linear  time  with  dynamic  programming  methods,  (.'yclic  probabilistic  decision  graphs 
can  be  solved  in  a  similar  way  in  polynomial  time  with  Markov  decision  algorithms. 
(For  very  risk-averse  agents  and  some  cyclic  probabilistic  ilecision  graphs,  however,  the 
agent  can  become  so  "scared"  of  the  risk  that  not  reaching  a  goal  state  liei'omes  as 
preferable  as  reaching  a  goal  state  even  if  the  planning  problem  is  solvable.) 

Figure  2.‘1  summarizes  tlie  worst  ra.se  run-»  imes  of  t  he  algorithms  that  we  iiav<*  discussi'd 
in  this  paper.  .Mthough  we  primarily  iliscussed  dynamic  programming  algorithms,  our 
approach  to  risk-sensitive  planning  <-an  be  usetl  to  augment  other  risk-.u'ulral  proba¬ 
bilistic  planning  algorithms  as  well.  .After  the  planning  problem  has  been  t  ransformetl. 
one  can  use  probabilistic  A I  planning  methods  or.  alternatively,  dynamic  programming 
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methods  on  the  transformed  planning  problem  to  determine  optimal  (or  good)  plans  for 
the  original,  risk-sensitive  planning  problem.  For  risk-seeking  agents,  one  can  use  amj 
planning  method  on  the  transformed  planning  problem  that  maximizes  (or  satisfices) 
expected  total  reward  (for  example,  the  planner  of  [Smith,  1988])  or,  equivalently,  the 
probability  of  goal  achievement  (for  example,  the  planner  of  [Kushmerick  et  a/.,  1993]) 
to  determine  an  optimal  (or  satisficing)  plan.  The  better  a  plan  is  for  the  transformed 
planning  problem,  the  better  it  is  for  the  original  planning  problem  as  well.  Since  our 
approach  allows  one  to  use  existing  probabilistic  planners  unchanged  for  risk-seeking 
planning  and  —  depending  on  the  planner  —  maybe  for  risk-averse  planning  as  well, 
it  extends  their  functionality.  Although  the  derivation  of  the  transformation  requires 
some  knowledge  of  utility  theory  and  Markov  decision  theory,  the  transformation  it¬ 
self  is  very  simple  and  can  be  applied  without  any  understanding  of  the  formalisms 
involved. 
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